feat(ci): save cache on `build-main` workflows and only restore cache on `docker-build-and-push-main` workflows #4865

youtalk · 2024-06-13T06:52:29Z

Description

fix(docker): fix CCACHE_DIR in Dockerfile #4863

The reproducible-containers/buildkit-cache-dance action introduced in #4854, unfortunately, might not support extracting the cache from the docker buildx bake and only supports docker build.

I confirmed that replacing docker/bake-action with docker/build-and-push-action as per the official sample code does indeed enable the cache to work.
https://github.com/reproducible-containers/buildkit-cache-dance?tab=readme-ov-file#examples

Therefore, in this PR, we will create a cache by running docker/build-and-push-action in the build-main and build-main-self-hosted workflows, which are scheduled to run daily, and in the sequentially executed docker-build-and-push-main and docker-build-and-push-main-self-hosted workflows, we will run docker/bake-action as in the past and only load the cache.

It turns out that the cache was effective in #4854 because I forgot to delete the cache from the commit during my experiment, which caused the cache to be effective. I apologize for the oversight.

Tests performed

Using this commit to temporarily run the build-main workflows in this PR, we confirmed that the execution time from the second run onwards was drastically reduced from the previous 2.5 hours to 1 hour.

first try (without cache): https://github.com/autowarefoundation/autoware/actions/runs/9495072771/usage
second try (with cache): https://github.com/autowarefoundation/autoware/actions/runs/9506736727/usage

https://autowarefoundation.github.io/autoware-ci-metrics/

Effects on system behavior

Not applicable.

Interface changes

Pre-review checklist for the PR author

The PR author must check the checkboxes below when creating the PR.

I've confirmed the contribution guidelines.
The PR follows the pull request guidelines.

In-review checklist for the PR reviewers

The PR reviewers must check the checkboxes below before approval.

The PR follows the pull request guidelines.

Post-review checklist for the PR author

The PR author must check the checkboxes below before merging.

There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Signed-off-by: Yutaka Kondo <[email protected]>

youtalk · 2024-06-13T07:05:38Z

The workflow namings are unclear, so I would like to refactor and rename them in another PR.
ref. #4747

Signed-off-by: Yutaka Kondo <[email protected]>

This reverts commit adced0d.

This reverts commit 7004dbe.

This reverts commit e6d62e4.

youtalk · 2024-06-14T00:22:43Z

Unfortunately, this PR still has an issue where the docker-build-and-push-main workflows cannot restore the cache, even though a fully matching cache is already available. However, since this issue can be resolved in a separate PR, we would like to consider this PR is ready for review for now.

Cache not found for input keys: cache-amd64-cuda-0187c3f0f9a23c6232cfe50f211e9deee563e665bcd440c1d921e03bb67898cc, cache-amd64-cuda-, cache-amd64-

https://github.com/autowarefoundation/autoware/actions/runs/9500467708/job/26183604143#step:6:13

https://github.com/autowarefoundation/autoware/actions/caches

xmfcx · 2024-06-16T17:36:38Z

Unfortunately, this PR still has an issue where the docker-build-and-push-main workflows cannot restore the cache, even though a fully matching cache is already available. However, since this issue can be resolved in a separate PR, we would like to consider this PR is ready for review for now.

This is happens because the workflow trigger of schedule isolates the cache.
Only scheduled build-main can get the cache of another scheduled build-main job, nothing else.
I guess not even workflow_dispatch event can reach it.

See: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

For this to work as intended, docker-build-and-push-main should maintain its own cache.

Also

instead of: cache-${{ inputs.platform }}-${{ inputs.name }}-${{ hashFiles('autoware.repos') }}
I propose: ccache-build-main-${{ runner.arch }}-humble-${{ github.sha }}

Reasons:

We don't need ${{ inputs.platform }} because ${{ runner.arch }} gives the similar result by using more generic variables.
- Possible values are X86, X64, ARM, or ARM64.
${{ inputs.name }} is cuda or no-cuda
- We should only generate the cache for cuda version. Because no-cuda packages are a subset of cuda packages.
- no-cuda version can restore and use the same cache.
Strictly separate actions/cache/save@v4 and actions/cache/restore@v4 actions instead of just using actions/cache@v4 action.
- This way we know who and when updates the cache and who just uses it.

The reason for -${{ github.sha }} is to know where the cache was created from.

restore-keys: |
  ccache-build-main-${{ runner.arch }}-humble-

Will fetch the most fresh version anyways. As long as it is unique, it will be enough.

@youtalk -san, I know you are busy this week and I would like to implement all I've mentioned until wednesday, is that OK?

youtalk · 2024-06-17T08:09:56Z

@xmfcx It's my pleasure! Please go ahead.

… on `docker-build-and-push-main` workflows (autowarefoundation#4865) * fix ccache_dir Signed-off-by: Yutaka Kondo <[email protected]> * update build-main Signed-off-by: Yutaka Kondo <[email protected]> * restore only Signed-off-by: Yutaka Kondo <[email protected]> * reorder Signed-off-by: Yutaka Kondo <[email protected]> * MUST REVERT Signed-off-by: Yutaka Kondo <[email protected]> * add cache-tag-suffix arg Signed-off-by: Yutaka Kondo <[email protected]> * rename Signed-off-by: Yutaka Kondo <[email protected]> * Revert "MUST REVERT" This reverts commit adced0d. * Revert "Revert "MUST REVERT"" This reverts commit 7004dbe. * Revert "Revert "Revert "MUST REVERT""" This reverts commit e6d62e4. --------- Signed-off-by: Yutaka Kondo <[email protected]>

youtalk added 6 commits June 13, 2024 11:11

fix ccache_dir

1ead255

Signed-off-by: Yutaka Kondo <[email protected]>

update build-main

cb79750

Signed-off-by: Yutaka Kondo <[email protected]>

restore only

3b83f3d

Signed-off-by: Yutaka Kondo <[email protected]>

reorder

637f2ea

Signed-off-by: Yutaka Kondo <[email protected]>

MUST REVERT

adced0d

Signed-off-by: Yutaka Kondo <[email protected]>

add cache-tag-suffix arg

a33f401

Signed-off-by: Yutaka Kondo <[email protected]>

youtalk requested a review from oguzkaganozt as a code owner June 13, 2024 06:52

youtalk marked this pull request as draft June 13, 2024 06:52

youtalk self-assigned this Jun 13, 2024

youtalk added 4 commits June 13, 2024 21:39

rename

02ce9d3

Signed-off-by: Yutaka Kondo <[email protected]>

Revert "MUST REVERT"

7004dbe

This reverts commit adced0d.

Revert "Revert "MUST REVERT""

e6d62e4

This reverts commit 7004dbe.

Revert "Revert "Revert "MUST REVERT"""

5038c22

This reverts commit e6d62e4.

youtalk marked this pull request as ready for review June 14, 2024 00:22

youtalk requested review from xmfcx and mitsudome-r June 14, 2024 00:23

youtalk added the type:ci Continuous Integration (CI) processes and testing. label Jun 14, 2024

mitsudome-r approved these changes Jun 14, 2024

View reviewed changes

youtalk merged commit 7738b5a into main Jun 14, 2024
19 checks passed

youtalk deleted the upstream-docker-build-action branch June 14, 2024 05:57

tier4-autoware-public-bot bot mentioned this pull request Jun 15, 2024

chore: sync upstream tier4/autoware#5

Open

This was referenced Jul 4, 2024

Only cuda jobs generate empty cache on build-main and build-main-self-hosted workflows #4944

Closed

fix(docker): remove build main action #4747

Closed

youtalk mentioned this pull request Jul 17, 2024

fix(ci): fix Docker image organization #5002

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ci): save cache on `build-main` workflows and only restore cache on `docker-build-and-push-main` workflows #4865

feat(ci): save cache on `build-main` workflows and only restore cache on `docker-build-and-push-main` workflows #4865

youtalk commented Jun 13, 2024 •

edited

Loading

youtalk commented Jun 13, 2024 •

edited

Loading

youtalk commented Jun 14, 2024

xmfcx commented Jun 16, 2024

youtalk commented Jun 17, 2024

feat(ci): save cache on build-main workflows and only restore cache on docker-build-and-push-main workflows #4865

feat(ci): save cache on build-main workflows and only restore cache on docker-build-and-push-main workflows #4865

Conversation

youtalk commented Jun 13, 2024 • edited Loading

Description

Tests performed

Effects on system behavior

Interface changes

Pre-review checklist for the PR author

In-review checklist for the PR reviewers

Post-review checklist for the PR author

youtalk commented Jun 13, 2024 • edited Loading

youtalk commented Jun 14, 2024

xmfcx commented Jun 16, 2024

youtalk commented Jun 17, 2024

feat(ci): save cache on `build-main` workflows and only restore cache on `docker-build-and-push-main` workflows #4865

feat(ci): save cache on `build-main` workflows and only restore cache on `docker-build-and-push-main` workflows #4865

youtalk commented Jun 13, 2024 •

edited

Loading

youtalk commented Jun 13, 2024 •

edited

Loading